Processing archive content based on hierarchical classification levels

ABSTRACT

Processing archive content based on hierarchical classification levels is performed by indexing a first newly archived file of a current set of archive content at a first classification level defining a plurality of types of metadata to be derived, wherein the first classification level is selected from a hierarchy of classification levels having at least two classification levels, indexing a second newly archived file of the current set of archive content at a second classification level, wherein the second classification level includes fewer types of metadata than the first class of metadata, and flagging the second newly archived file of the current set of archive content. By classifying and flagging the second newly archived file differently than the first newly archived file, the system can easily retrieve such data at a later instance to update and/or change the data, re-index the data, and the like.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation of U.S. patent applicationSer. No. 13/313,348, filed Dec. 7, 2011.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates in general to file storage systems, andmore particularly, to processing archive content.

2. Description of the Related Art

A storage management application, such as IBM® Tivoli® Storage Manager®(TSM) (registered to International Business Machines Corporation ofArmonk, N.Y.), may be implemented on a storage management server. Thestorage management application manages storage requirements for aplurality of client nodes that are coupled to the storage managementserver via a network. The storage management application creates andmanages a repository for data and programs that are backed up, archived,migrated, or otherwise copied from the client nodes to the storagemanagement server.

The storage management server stores data objects, such as files, in oneor more storage pools and uses a database stored in the storagemanagement server for tracking information about the stored dataobjects. In this regard, an index on metadata associated with the dataobjects may be included. To locate certain archived files, the archiveis mined via queries made on the index.

SUMMARY OF THE INVENTION

Usefulness of the index in locating certain archived files depends onthe type, quantity, variety, and quality of the metadata extracted fromthe archived files. In some instances during archiving, the extractedmetadata is found to be corrupted and/or metadata cannot be extractedfrom one or more of the archived file. In other instances, a processingerror occurs during archiving and metadata is not extracted from the oneor more archived files. Improved methods and systems are now providedthat are useful for identifying such archived files so that if archivingtechnology improves, metadata may be extracted from these identifiedarchived files.

In an embodiment, by way of example only, a method includes indexing afirst newly archived file of a current set of archive content at a firstclassification level, the first classification level defining aplurality of types of metadata to be derived from the current set ofarchive content, wherein the first classification level is selected froma hierarchy of classification levels having at least two classificationlevels, indexing a second newly archived file of the current set ofarchive content at a second classification level, wherein the secondclassification level is selected from the hierarchy of classificationlevels and defines a second type of metadata is to be derived from thecurrent set of archive content, and the second class of metadataincludes fewer types of metadata than the first class of metadata, andflagging the second newly archived file of the current set of archivecontent.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the invention will be readilyunderstood, a more particular description of the invention brieflydescribed above will be rendered by reference to specific embodimentsthat are illustrated in the appended drawings. Understanding that thesedrawings depict only typical embodiments of the invention and are nottherefore to be considered to be limiting of its scope, the inventionwill be described and explained with additional specificity and detailthrough the use of the accompanying drawings, in which:

FIG. 1 is a schematic of a network computing environment, according toan embodiment;

FIG. 2 is a schematic of a system that indexes information that isingested into archival storage, according to an embodiment; and

FIG. 3 is a flow diagram of a method of processing archive content,according to an embodiment.

DETAILED DESCRIPTION OF THE DRAWINGS

The illustrated embodiments below relate to processing archive contentbased on hierarchical classification levels. The embodiments includeprocessing archive content by indexing a first newly archived file of acurrent set of archive content at a first classification level, thefirst classification level defining a plurality of types of metadata tobe derived from the current set of archive content, wherein the firstclassification level is selected from a hierarchy of classificationlevels having at least two classification levels, indexing a secondnewly archived file of the current set of archive content at a secondclassification level, wherein the second classification level isselected from the hierarchy of classification levels and defines asecond type of metadata is to be derived from the current set of archivecontent, and the second class of metadata includes fewer types ofmetadata than the first class of metadata, and flagging the second newlyarchived file of the current set of archive content. By classifying andflagging the second newly archived file differently than the first newlyarchived file, the system can easily retrieve such data at a laterinstance to update and/or change the data, re-index the data, and thelike. Such methods can be retro-fitted into existing systems and areuseful for archiving files of various types of file storage systems,such as clustered storage systems.

FIG. 1 illustrates a computing environment in which embodiments of theinvention are implemented. An archive server 2 maintains archive objectsin archive storage 4. The archived objects may originate from clientsystems 6 a, 6 b, 6 c, where the clients 6 a, 6 b, 6 c communicateobjects to the archive server 2 over a network 10. In accordance with anembodiment, the archive server 2 includes an archive program whichperforms archive related operations, an archive database maintainingrecords providing information on archived objects, and a retentionprotection setting controlling the extent to which the archive programpermits users to remove or modify information in the database andarchived objects from the archive storage 4. All archive relatedoperations are managed by the archive program to ensure compliance witha configured archival policy. The archive database may include an objecttable having an entry (record) for each archived object providinginformation on the object and an expiration table having one entry(record) for each initiated or expired retention period running withrespect to one archived object.

The archive server 2 may be implemented as a clustered file system andcomprises one or more nodes. The archive server 2 is a server classmachine, in an embodiment, and the client systems 6 a, 6 b, 6 c maycomprise any computing device known in the art, such as a server classmachine, mainframe, workstation, desktop computer, handheld computer,etc. The data archived from the client systems 6 a, 6 b, 6 c maycomprise data in a database, application program data, files in afilesystem, etc. The archival storage 4 may comprise any mass storagedevice known in the art, such one or more interconnected disk drivesconfigured as a Redundant Array of Independent Disks (RAID), Just aBunch of Disks (JBOD), Direct Access Storage Device (DASD), as a tapestorage device, e.g., a tape library, or etc. The network 10 maycomprise any interface between storage and a host known in the art, suchas a network connection (e.g., Ethernet, wireless ethernet, FibreChannel, etc.) or any other network or storage transfer protocol knownin the art.

FIG. 2 is a schematic of a system 200 that indexes information that isingested into archival storage (e.g., archive storage 4), in anembodiment. The system 200 is a node of the archive server 2 andgenerally includes a file system 202, a file server 204, an extendedattribute handler 206, an event handler 208, a document handler 210, anevent log handler 212, an ingestion handler 214, an index handler 216,and an index server 218. The file system 202 resides on the node andincludes files. The file server 204 communicates with the file system202 and presents files gathered from the file system 202 to the extendedattribute handler 206 to associate additional metadata to the files.According to an embodiment, the file server 204 is included in thearchive server 2. In an embodiment, the metadata can includenon-standard attributes, such as user-defined metadata (i.e.,department, author or other similar data, (name,value) pairs of typestring, numeric, Boolean, date), archive extended attributes (e.g.,service class, md5sum, retention time, etc.), archive indexing extendedattributes (e.g., index level, index epoch, index time, index core,etc.), or other types of attributes.

The event handler 208, the document handler 210, event log handler 212,ingestion handler 214, and index handler 214 can be included as part ofthe backup server program 114. In an embodiment, the event handler 208handles inputs received by the backup server program 114. The documenthandler 210 is configured to store metadata values into extendedattributes and initiate index processing by the index handler 216. Theevent log handler 212 captures file system events for later processing,including ingesting and indexing and the like. The ingestion handler 214is configured to coordinate the processing required for trapped filesystem events. For example, the ingestion handler 214 initiatesprocessing of newly archived files by the document handler 210. Inanother example, the ingestion handler 214 initiates processing ofdeleted archived files by the index handler 216. The index handler 216is configured to assimilate all information needed to add/update entriesin the index or delete entries from the index. The index server 218efficiently stores and retrieves indexing information on extractedmetadata and text to/from an underlying index. In another embodiment,the archive server 2 includes additional nodes without the index server218.

During operation, the system 200 uses the metadata to index and searchfor archived content. FIG. 3 is a method of processing archive content,according to an embodiment. In an embodiment, a classification level forwhich metadata is derived from a current set of archive content is setat a first classification level, step 302. In this regard, theclassification level is selected from a hierarchy of classificationlevels. Each level includes different types of metadata that is derivedfrom the archive content. In an example, the hierarchy of classificationlevels includes four levels—level 3, level 4, level 5, and level 6. Forlevel 3, the archive content is not indexed. Level 4 can includeindexing archive content according to file system attributes, archiveextended attributes, and user metafile metadata. Level 5 can includeindexing archive content according to the attributes listed in level 4and, in addition, extracted metadata (e.g., text extracted from contentsof a file, author/title of a file, and the like). Level 6 can includeindexing archive content according to the attributes listed in level 5and in addition, extracted metadata with highlighting. In otherembodiments, more or fewer classification levels are included and/oreach level indexes the archive content differently. The firstclassification level is a highest level enabled by the system or user.

With additional reference to FIG. 2, according to an embodiment, thecurrent set of archive content comprises a file that is ingested by thefile server 206 and is to be archived (referred to herein as “newlyarchived file”). Before the newly archived file is indexed, the fileserver 206 presents the newly archived file to the extended attributehandler 206 to associate certain metadata to the newly archived fileprior to being presented to the file system 202, in an embodiment. Inanother embodiment, the newly archived file is presented directly to thefile system 202. The event handler 208 detects the presentation of thenewly archived file to the file system 202 and logs the presentation asan event on an event log. The ingestion handler 214 obtains a change logfrom the event log handler 212 and applies a given policy to the filesystem 202. The document handler 210 sets extended attributes accordingto the policy and logs on an index any changes made to the file system202 as a result of the application of the policy. For example, thedocument handler 210 makes insertions and/or updates to the index. Theindex handler 216 assimilates all information needed related to thenewly archived file to add/update entries in the index or delete entriesfrom the index on the index server 218, depending on the selectedclassification level.

One or more of the newly archived files of the archive content isindexed at the first classification level, step 304. In an embodiment inwhich the document handler 210 sets the first classification level atLevel 4, the index handler 216 assimilates metadata to be indexed andsets indexing of extended attributes of the newly archived file. Inanother embodiment in which the document handler 210 sets the firstclassification level at Level 5, the index handler 216 assimilatesmetadata to be indexed and sets indexing of extended attributes of thenewly archived file and the index server 218 extracts metadata. If thefirst classification level is at Level 6, the index handler 216assimilates metadata to be indexed and sets indexing of extendedattributes of the newly archived file, and the index server 218 extractsmetadata and stores extracted text in the index.

In an embodiment, metadata included in a file of the archive contentcannot be extracted or contains extraction errors or processing errors.Hence, the file is indexed at a second classification level, step 306.The second classification level is a classification level that includesfewer types of metadata derived from the archive content than thosetypes derived for the first classification level. In an embodiment, thesecond classification level is the highest level from which metadata issuccessfully generated or derived from the archive content.

Next, the file indexed at the second classification level is flagged,step 308. An identifier or tag is associated with the current archivecontent. In an example, if an attempt to index the newly archived fileat the first classification level (e.g., Level 5 or Level 6) isunsuccessful (e.g., an extraction error or processing error occurs) theindex handler 216 retries indexing at a lower classification level(e.g., Level 5, if the first classification level is level 6, or Level4, if the first classification level is Level 5). The index handler 216flags the newly archived file with a change in classification level(e.g., from Level 5 to Level 4). In this way, the flagged file of thecurrent archive content can be identified for re-indexing when metadatageneration technology improves. After step 308, a next set of archivecontent is indexed, step 310.

In an embodiment, the archive content is re-indexed, step 312.Re-indexing occurs in response to a request to index the archive contentat a new classification level, or additions/changes to user definedmetafile metadata, or a retroactive policy scan. In an embodiment inwhich re-indexing occurs by request, the ingestion handler 214 scans thefile system 202 for a set of archive content previously indexed at alevel classification lower than the new classification level, and theindex server 218 extracts the metadata from the previously-indexed setof archive content at the new classification level. In anotherembodiment in which re-indexing occurs by addition or changes to userdefined metafile metadata, the extended attribute handler 206 signalschanges or additions to (name,value) pairs by setting new extendedattributes for the newly archived files on a per file basis, whichcauses the ingestion handler 214 to scan the file system 202 for a setof archive content previously indexed without the new extendedattributes and the index handler 216 to index the previously-indexed setof archive content with the new extended attributes. If re-indexingoccurs by a retroactive policy scan, the document handler 210 signals achange by setting new extended attributes for the newly archived fileson a per file bases, which causes the ingestion handler 214 to scan thefile system 202 for a set of archive content previously indexed withoutthe new extended attributes and the index handler 216 to index thepreviously-indexed set of archive content with the new extendedattributes.

Indexing archive content at a highest classification level enabled by auser or system or at a highest classification level at which metadata issuccessfully generated or derived from the archive content and flaggingthe content indexed at a classification level lower than the highestenabled classification level allows such content to be retrieved laterfor re-indexing. In particular, indexing at a new classification levelcan be applied retroactively to files that have already been indeed.Additionally, the above-described methods and systems maintainconsistency between the index and archive content as files are indexedand/or re-indexed.

As will be appreciated by one of ordinary skill in the art, aspects ofthe present invention may be embodied as a system, method, or computerprogram product. Accordingly, aspects of the present invention may takethe form of an entirely hardware embodiment, an entirely softwareembodiment (including firmware, resident software, micro-code, etc.) oran embodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module,” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer-readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer-readable medium(s) may beutilized. The computer-readable medium may be a computer-readable signalmedium or a physical computer-readable storage medium. A physicalcomputer readable storage medium may be, for example, but not limitedto, an electronic, magnetic, optical, crystal, polymer, electromagnetic,infrared, or semiconductor system, apparatus, or device, or any suitablecombination of the foregoing. Examples of a physical computer-readablestorage medium include, but are not limited to, an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk,RAM, ROM, an EPROM, a Flash memory, an optical fiber, a CD-ROM, anoptical storage device, a magnetic storage device, or any suitablecombination of the foregoing. In the context of this document, acomputer-readable storage medium may be any tangible medium that cancontain, or store a program or data for use by or in connection with aninstruction execution system, apparatus, or device.

Computer code embodied on a computer-readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wired, optical fiber cable, radio frequency (RF), etc., or any suitablecombination of the foregoing. Computer code for carrying out operationsfor aspects of the present invention may be written in any staticlanguage, such as the “C” programming language or other similarprogramming language. The computer code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, or communication system, including, but notlimited to, a local area network (LAN) or a wide area network (WAN),Converged Network, or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).

Aspects of the present invention are described above with reference toflow diagrams and/or block diagrams of methods, apparatus (systems) andcomputer program products according to embodiments of the invention. Itwill be understood that each block of the flow diagrams and/or blockdiagrams, and combinations of blocks in the flow diagrams and/or blockdiagrams, can be implemented by computer program instructions. Thesecomputer program instructions may be provided to a processor of ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer orother programmable data processing apparatus, create means forimplementing the functions/acts specified in the flow diagram and/orblock diagram block or blocks.

These computer program instructions may also be stored in acomputer-readable medium that can direct a computer, other programmabledata processing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flow diagram and/or blockdiagram block or blocks. The computer program instructions may also beloaded onto a computer, other programmable data processing apparatus, orother devices to cause a series of operational steps to be performed onthe computer, other programmable apparatus or other devices to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flow diagram and/orblock diagram block or blocks.

The flow diagrams and block diagrams in the above figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflow diagrams or block diagrams may represent a module, segment, orportion of code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flow diagrams, andcombinations of blocks in the block diagrams and/or flow diagram, can beimplemented by special purpose hardware-based systems that perform thespecified functions or acts, or combinations of special purpose hardwareand computer instructions.

What is claimed is:
 1. A method of processing archive content, themethod comprising: indexing a first newly archived file of a current setof archive content at a first classification level, the firstclassification level defining a plurality of types of metadata to bederived from the current set of archive content, wherein the firstclassification level is selected from a hierarchy of classificationlevels having at least two classification levels; indexing a secondnewly archived file of the current set of archive content at a secondclassification level, wherein the second classification level isselected from the hierarchy of classification levels and defines asecond type of metadata is to be derived from the current set of archivecontent, and the second class of metadata includes fewer types ofmetadata than the first class of metadata; flagging the second newlyarchived file of the current set of archive content; locating allflagged archive content at a later time; re-indexing the flagged archivecontent at the second classification level of the hierarchy ofclassification levels at the later time; and setting a classificationlevel to a third classification level of the hierarchy of classificationlevels in which no metadata is derived from archive content.
 2. Themethod of claim 1, wherein the hierarchy levels include a classificationlevel in which file system attributes, general parallel file systemarchive extended attributes, and user metafile metadata are derived fromarchive content.
 3. The method of claim 1, wherein the hierarchy levelsinclude a classification level in which file system attributes, generalparallel file system archive extended attributes, user metafilemetadata, and extracted metadata are derived from archive content. 4.The method of claim 1, wherein the hierarchy levels include aclassification level in which file system attributes, general parallelfile system archive extended attributes, user metafile metadata, andextracted metadata with highlighting are derived from archive content.5. The method of claim 1, further comprising: detecting a retroactivepolicy scan, before the step of re-indexing.
 6. The method of claim 1,wherein the step of indexing the second newly archived file of thecurrent archive content includes indexing the second newly archived filewith extraction errors.
 7. The method of claim 1, wherein the step ofindexing the second newly archived file of the current archive contentincludes indexing the second newly archived file due to processingerrors.
 8. The method of claim 1, wherein indexing a second newlyarchived file includes initially attempting to index the second newlyarchived file at the first classification level, and indexing the secondnewly archived file at the second classification level, if the secondnewly archived file includes archive content required for extraction atthe first classification level that cannot be extracted.
 9. The methodof claim 1, wherein indexing a second newly archived file includes:signaling changes to user defined new extended attributes associatedwith the newly archived file; scanning the file system for archivecontent previously indexed without the new extended attributes, inresponse to the changes; and re-indexing the flagged newly archived filewith the new extended attributes.
 10. The method of claim 5, whereindetecting the retroactive policy scan includes initiating the computercode for locating all the flagged archive content.